The use of automated problem detection and remediation systems enable the services support industry to transition from reactive support to proactive and preemptive support. The automated problem detection and remediation system may leverage machine consumable intellectual capital (IC) rules (e.g., software code modules) that detect and solve problems in customer devices. In some examples, problem detection engines may leverage IC rules to detect problems in customer device support data, and may run thousands of times per day. The engines may process data from many different types of devices, with each device configured differently per the customer's network.
Creating machine consumable IC rules to detect specific issues presents challenges due to the multitude of potential combinations of devices and configurations that may not be envisioned when creating the IC rules. Even if the logic of the IC rule seems correct, the IC rule may fail to work properly on data generated by a device that is configured slightly differently than the IC author expected. The unexpected configuration may cause the execution engine processing the IC modules against the input data to crash, or produce a false positive or false negative result.
Continuous integration allows for a number of static tests with a static input dataset whenever a new code change is performed. Unit testing is a software testing technique where individual functions are run against input data sets and their outputs are checked for correctness. However, the input data for both continuous integration and unit testing is typically static and may not reflect the configuration of devices as they are implemented in the real world.